Setting general connection properties

This section describes how to configure general connection properties. For an explanation of how to configure advanced connection properties, see Setting advanced connection properties.

To define the general connection properties:

1. Click the Manage Endpoint Connections toolbar button.

   The Manage Endpoints Connections dialog box opens.

2. Click the New Endpoint Connection toolbar button.

   The Name, Description, Type and Role fields are displayed on the right.

3. In the Name field, specify a display name for the endpoint.
4. In the Description field, optionally type a description for the MapR Streams endpoint.
5. Select Target as the endpoint Role.

6. Select MapR Streams as the endpoint Type.

   The dialog box is divided into General and Advanced tabs.

7. In the Cluster name field, specify the MapR Streams cluster name.

   To check if the cluster is valid, execute the following command in a Linux shell:

   hadoop fs -ls maprfs://cluster_name/

   If there is a directory listing for the cluster, the cluster is valid.

8. In the Data Publishing section, set the following properties:

   1. In the Publish the data to field, choose one of the following:

      • Specific topic - to publish the data to a single topic. Type the stream name followed by a colon and the topic name.

        Example:

        MapRStreamName:MapRTopicName

      • Specific topic for each table - to publish the data to multiple topics corresponding to the source table names.

        In the Stream name field, enter the stream name only (i.e. stream-name as opposed to stream name:topic-name).

      Information note

      If the topics do not exist, the stream must be configured with autocreate=true to enable Replicate to create the topics during runtime. Otherwise, the task will fail.

   2. From the Partition strategy drop-down list, field, select either Random or By message key. If you select Random, each message will be written to a randomly selected partition. If you select By message key, messages will be written to partitions based on the selected By message key (described below).

   3. From the Message key drop-down list, field, select one of the following:

      Information note

      Note  The message key is represented as a string, regardless of the selected data message format (JSON/Avro).

      • None - To create messages without a message key.

      • Schema and table name - For each message, the message key will contain a combination of schema and table name (e.g. "dbo+Employees").

        When By message key is selected as the Partition strategy, messages consisting of the same schema and table name will be written to the same partition.

      • Primary key columns - For each message, the message key will contain the value of the primary key column.

        When By message key is selected as the Partition strategy, messages consisting of the same primary key value will be written to the same partition.

10. In the Message Properties section, set the following properties:

    1. Choose JSON or Avro as the message format.

       Information note

       Qlik provides an Avro Message Decoder SDK for consuming Avro messages produced by Qlik Replicate. You can download the SDK as follows:

       1. Go to Product Downloads.

       2. Select Qlik Data Integration.

       3. Scroll down the Product list and select Replicate.

       4. In the Download Link column, locate the QlikReplicate_<version>_Avro_Decoder_SDK.zip file. Before starting the download, check the Version column to make sure that the version correlates with the Replicate version you have installed.

       5. Proceed to download the QlikReplicate_<version>_Avro_Decoder_SDK.zip file.

       For usage instructions, see Kafka Avro consumers API.

       An understanding of the Qlik envelope schema is a prerequisite for consuming Avro messages produced by Qlik Replicate. If you do not wish to use the SDK, see The Qlik Envelope for a description of the Qlik envelope schema.

    2. To publish the schema message (for the corresponding data message) to a topic, select the Use Schema Messages check box and then type the stream name followed by a colon and the topic name.

       Example:

       MapRStreamName:MapRTopicName

       This option is required if the message format is set to Avro since Avro-formatted messages can only be opened using the Avro schema.

       Information note

       It is strongly recommended not to publish schema messages to the same topic as data messages.

       Information note

       If the topics do not exist, the stream must be configured with autocreate=true to enable Replicate to create the topics during runtime. Otherwise, the task will fail.